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ABSTRACT 


This  report  documents  the  work  conducted  by  the  HIT  Lab  during  a  four-year  project  titled 
“Communicating  Situation  Awareness  in  Virtual  Environrnents.”  The  project  was  funded 
under  the  MURI  (Multi-disciplinary  University  Research  Initiative),  and  was  intended  as 
“spin-up”  funding  to  allow  the  Lab  to  achieve  critical  mass  and  momentum.  As  such,  the 
goals  of  both  the  fund  providers  and  fund  recipients  were  successfully  achieved.  Over  the 
course  of  the  project  over  30  experiments  were  conducted  resulting  in  76  publications. 
Support  was  provided  for  approximately  20  students,  resulting  in  9  theses  and 
dissertations.  A  multi-disciplinary  workshop  was  conducted,  and  there  were  active 
collaborations  between  researchers  in  this  lab,  with  other  labs,  with  government  agencies 
and  with  commercial  companies.  The  benefits  of  this  collaboration  are  beginning  to  take 
effect. 

The  focus  of  the  research  effort  was  tightened  during  the  last  year  of  the  project  to  address 
five  key  areas:  1)  Motion  sickness  in  users  of  virtual  environments  2)  Collaborative  multi¬ 
crew  virtual  environments  3)  Situation  awareness  in  spatialized  auditory  and  spatialized 
visual  environments  4)  Human  factors  assessment  of  virtual  interface  tools  5)  Human 
factors  assessment  of  the  virtual  motion  controller.  The  results  from  experiments 
investigating  four  of  these  five  areas  are  presented  in  this  report,  along  with  a  cumulative 
list  of  all  of  the  publications. 
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1. 


Research  Objectives 


The  original  research  objectives  of  the  project  are  outlined  in  previous  interim  reports.  A 
set  of  new  research  objectives  were  worked  out  between  February  and  May  1996,  under 
the  auspices  of  the  AFOSR  contract  manager,  Dr.  John  Tangney,  and  the  new  Principal 
Investigator' ,  Dr.  Max  Wells.  The  general  mbric  of  the  program  -  the  communication  of 
situation  awareness  in  virtual  environments  -  remained  the  same.  However  the  focus  was 
sharpened  to  address  the  questions: 

•  Why  should  the  Air  Force  invest  in  VR,  i.e.  what  are  the  benefits? 

•  What  enabling  technologies  and  know-how  are  required  to  realize  these  benefits? 
It  was  decided  to  concentrate  on  the  five  areas  listed  below.  This  report  documents  work 
conducted  during  the  fourth  and  final  year  of  the  project,  in  four  of  these  five  areas.  Work 
in  one  of  the  areas,  spatialized  auditory  and  visual  environments,  was  discontinued  during 
the  final  year  of  the  project  due  to  the  sudden  departure  of  one  of  the  co-PIs. 

1.1.  Motion  Sickness  in  Users  of  Virtual  Environments 

The  objectives  of  this  effort  are:  to  improve  the  theory  of  causation  of  motion  sickness 
(MS)  in  VR,  to  improve  the  measures  of  MS,  to  determine  system  factors  that  increase 
incidence  of  MS  and  to  study  methods  of  configuring  systems  and  training  users  to 
minimize  MS. 

1.2.  The  Virtual  Pilot  (Collaborative,  multi-crew  virtual 
environments) 

The  objective  of  this  research  effort  is  to  create  a  testbed  in  which  to  implement  and  test  the 
concept  and  utility  of  geographically  dispersed  teams  working  together  on  collaborative 
tasks  using  computer-mediated  communication  technology,  including  virtual  reality.  This 
project  has  generated  industry  interest  under  the  SHARE  (Shared  Augmented  Reality) 
consortium  of  Boeing,  GEC,  Microvision,  and  the  HIT  Lab. 

1.3.  Situation  Awareness  in  Spatialized  Auditory  and  Spatialized 
Visual  Virtual  Environments 

The  objective  of  this  research  is  to  determine  how  information  from  augmented  reality 
visual  and  auditory  displays  can  be  effectively  integrated  with  existing  cockpit  information 
to  enhance  situational  awareness.  Research  in  this  area  was  discontinued  under  AFOSR 
support  with  the  departure  of  Dr.  Woodrow  Barfield. 

1.4.  Human  Factors  Assessment  of  Virtual  Interface  Tools 

Among  the  major  human  factors  issues  surrounding  the  military  use  of  virtual 
environments  is  the  question  of  how  to  design  the  user  interface  so  as  to  minimize  its 
cognitive  load  while  maximizing  environment  effectiveness.  This  activity  evaluated  the 
usability  and  utility  of  a  series  of  immersive  interface  tools,  widgets,  and  metaphors.  The 
research  goal  is  to  determine  the  relative  merits  of  each  technique  for  a  given  task  and 
domain. 

1.5.  Human  Factors  Assessment  of  the  Virtual  Motion  Controller 
The  HIT  Lab  has  developed  a  device  -  the  Virtual  Motion  Controller  (VMC)  -  for 
locomoting  through  virtual  environments.  The  VMC  allows  hands-free  operation,  and  uses 
intuitive  foot  and  body  action  for  moving  through  virtual  environments.  Our  research 
objective  is  to  compare  the  VMC  with  some  other  methods  of  moving  through  VR  and 
quantify  the  benefits  of  using  this  new  interface  device. 


'  Dr  Wells  took  over  as  Prinicipal  Investigator  for  the  final  year  of  the  project,  effective 
June  1996. 


- 1- 


2.  Status  Of  The  Research  Effort 

2.1.  Motion  Sickness  in  Users  of  Virtual  Environments 

2. 1.1. Eye  Movement  Research 

Current  virtual  interfaces  imperfectly  simulate  the  motion  dynamics  of  the  real  world.  These 
imperfections  have  a  range  of  consequences  which  were  explored  in  a  Ph.D.  thesis  by 
Captain  Mark  Draper. 

Conflicting  visual  and  vestibular  cues  of  self-motion  are  believed  to  drive  physiological 
adaptations  and  simulator  sickness,  which  raises  significant  health  and  safety  issues 
surrounding  virtual  environment  exposure.  Research  investigated  the  nature  of  human 
physiological  adaptation  to  virtual  interfaces  through  a  detailed  study  of  the  vestibulo-ocular 
reflex  (VOR).  The  VOR  is  a  compensatory  eye  movement  response  that  functions  to  keep 
the  visual  scene  stabilized  on  the  retina  during  head  movements.  The  VOR  is  investigated 
because  of  its  propensity  to  adapt  to  visual-vestibular  sensory  rearrangements  and  because 
symptomology  during  VOR  adaptation  is  often  quite  similar  to  that  of  simulator  sickness. 

The  main  hypothesis  under  investigation  holds  that  certain  artifacts  of  virtual  interfaces 
drive  VOR  gain  and  phase  adaptation  processes.  Artifacts  investigated  included  system  time 
delays  (between  head  movement  initiation  and  visual  scene  response;  48  ms,  125  ms,  and 
250  ms)  and  virtual  image  scale-factor  changes  (e.g.,  scene  magnification  level:  0.5X, 
l.OX,  2.0X).  Measures  of  VOR  adaptation  included  gain  and  phase  response  changes  from 
baseline  levels,  while  simulator  sickness  metrics  included  oral  self  reports  and  a  post¬ 
exposure  questionnaire  (SSQ). 

Results  demonstrated  that  significant  VOR  gain  adaptation  magnitude  and  direction  resulted 
from  image  scale  factors  that  differed  from  l.OX  magnification  (Figure  1).  Simulator 
sickness  reports  were  also  affected  by  image  scale  factor  (Figure  2).  Time  delays  resulted 
in  statistically  significant  VOR  gain  decreases  and  increases  in  phase  lag,  though  simulator 
sickness  did  not  increase  with  increasing  time  delays. 

Information  obtained  from  these  studies  were  used  to  develop  preliminary  design 
guidelines  for  the  reduction  of  unwanted  interface-generated  effects  on  the  user  (Draper, 
1998). 
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2. 1.2.  The  Rest  Frame  Hypothesis 

The  following  abstract  describes  the  work  in  a  Ph.D.  thesis  by  Jerry  Prothero: 

Two  fundamental  human  factors  problems  limit  the  application  of  viitual  interfaces.  The 
first  is  the  lack  of  robust  goodness  measures  to  guide  design;  the  second  is  simulator 
Sickness.  The  rest  frame  hypothesis  (RFH)  is  introduced  to  address  these  problems.  The 
RFH,  based  on  the  observation  that  we  have  a  strong  perception  that  certain  things  are 
stationary,  suggests  that  selected  rest  frames  are  fundamental  to  spatial  perception. 
Presence,  the  sense  of  “being  somewhere”,  is  defined  in  terms  of  the  choice  of  selected  rest 
frame.  “Objective”  presence  measures  based  on  visual-inertial  nulling  are  introduced  in 
terms  of  this  definition.  Literature  is  reviewed  suggesting  that  presence  is  one  (but 
certainly  not  the  only)  measure  for  the  quality  of  an  interface.  A  technique  motivated  by  the 
RFH  is  introduced  for  reducing  simulator  sickness,  employing  visual  background 
manipulations.  Other  topics  discussed  include  the  application  of  foreground  occlusions  to 
head-mounted  displays.  (Prothero,  1998) 

2. 1. 3.  Intertial-Visual-Nulling 


Introduction 

The  sense  of  presence  (or  of  “being  somewhere”)  is  important  to  the  study  of  virtual 
interfaces  for  two  reasons.  The  first  reason  is  that  the  feeling  of  being  present  in  a 
computer-generated  space  is  characteristic  of  virtual  interfaces.  Hence,  any  attempt  to 
explain  the  psychology  of  virtual  interfaces  must  explain  presence.  The  second  reason  is 
that  there  is  evidence  that  presence  is  related  to  the  degree  to  which  an  interface  is  intuitive 
or  conveys  meaning.  Conversely,  a  good  presence  measure  may  be  one  means  to  assess 
the  quality  of  an  interface. 

Presence  has  typically  been  assessed  with  self-report  measures,  such  as  asking  participants 
to  rate  their  sense  of  presence  on  a  scale  of  1-7.  Such  measures  are  prone  to  several 
difficulties.  The  first  is  a  lack  of  sensitivity:  assigning  numbers  to  fine  gradations  in  mental 
state  is  not  a  task  humans  were  designed  for.  The  second  is  lack  of  consistency  across 
individuals.  Rating  scales  between  different  individuals  are  likely  to  be  assigned  in 
different  ways.  The  third  is  that  self-report  measures  for  a  given  condition  tend  to  depend 
on  what  it  is  compared  to.  Hence,  it  is  difficult  to  compare  data  across  experiments. 


-3- 


The  research  described  here  seeks  to  measure  presence  perceptually.  The  idea  is  that  if 
presence  reflects  the  degree  to  which  someone  is  "pulled  into"  a  visual  scene,  then  presence 
should  be  measurable  by  setting  up  an  experiment  in  which  the  visual  cues  conflict  with 
cues  from  the  external  environment.  The  degree  to  which  the  visual  cues  overwhelm  the 
external  cues  may  define  a  scale  for  presence. 

The  “cross-over”  measure  introduced  here  sets  up  a  conflict  between  inertial  and  visual 
self-motion  cues  in  the  horizontal  plane.  The  effectiveness  of  two  visual  conditions  was 
assessed  in  terms  of  their  relative  ability  to  overwhelm  conflicting  inertial  motion  cues.  The 
measure  consisted  of  finding  the  inertial  amplitude  below  which  participants  crossed-over 
to  perceiving  the  motion  implied  by  the  visual  scene,  even  though  they  were  trying  to 
follow  the  inertial  motion.  Inertial  and  visual  motion  were  restricted  to  the  horizontal  plane 
to  avoid  strong  inertial  cues  from  gravity. 

The  following  questions  about  this  inertial-visual  nulling  technique  were  addressed. 

•  Can  it  detect  a  predicted  manipulation  of  presence?  Specifically,  can  it  find  a 
difference  between  meaningful  and  random  visual  scenes? 

•  What  is  it's  test-retest  reliability? 

•  How  is  it  related  to  self-report  presence  measures? 

•  How  is  it  related  to  frame  dependency  measures? 

"Frame  dependency"  is  a  standard  psychological  term  referring  to  the  degree  to  which 
different  people  are  influenced  by  visual  over  conflicting  inertial  cues.  It  is  hypothesized 
that  between  participants,  a  good  presence  measure  should  correlate  with  frame 
dependency. 

Methods; 

There  were  12  adult  participants.  Participants  were  seated  upright  in  a  chair  which  could 
oscillate  in  the  horizontal  plane.  They  wore  a  Virtual  Research  VR4  HMD  with  a  48  deg 
FOV.  Head-tracking  was  not  used.  The  scene  was  displayed  to  both  eyes,  but  was  not 
stereoscopic.  In  order  to  keep  the  frame  rate  high  (60  fps),  the  resolution  was  lowered  to 
240x320  pixels.  Both  the  chair  and  the  visual  scene  were  oscillated  at  0.1  Hz.  The  visual 
amplitude  was  fixed  at  30  deg;  the  inertial  amplitude  was  systematically  varied  as  described 
below.  The  self-motion  implied  by  the  visual  oscillation  was  set  to  trail  the  self-motion 
implied  by  the  chair  motion  by  90  deg. 

In  each  of  two  sessions,  participants  were  exposed  to  two  conditions:  a  picture  of  a  scene 
from  Maui  (meaningful  condition)  or  the  same  set  of  pixels  randomized.  At  the  beginning 
of  each  session,  per  exposure  reported  presence  ratings  were  obtained  for  each  condition 
following  a  one-minute  period  in  which  participants  were  asked  to  look  around  and  gather 
their  impressions  of  the  scene. 

Trials  from  each  of  the  two  conditions  alternated  on  an  ABBA  pattern.  Before  each  trial, 
the  chair  was  started  from  rest  and  participants  were  asked  to  count  down  by  7's  for  25 
seconds  from  an  arbitrary  number,  with  their  eyes  closed.  This  served  to  distract  them 
from  "locking  in"  to  the  chair  motion.  They  were  then  asked  to  count  down  by  7's  for  an 
additional  25  seconds  with  their  eyes  open.  This  gave  the  visual  cues  time  to  gain  strength. 
Finally,  they  were  asked  to  stop  counting  and  to  signal  by  switching  a  toggle  the  perceived 
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right/left  extremes  of  the  chair  motion,  while  watching  the  visual  scene.  We  recorded  the 
mean  phase  angle  of  the  participant's  right/left  responses  from  the  intended  inertial 
endpoint,  and  from  the  right/left  endpoints  implied  by  the  visual  stimulus. 

At  high  inertial  amplitudes,  most  participants  had  little  difficulty  correctly  following  the 
inertial  motion  while  watching  the  visual  scene.  At  lower  amplitudes,  most  participants  fell 
into  "visual  capture",  signaling  the  visual  endpoints  of  the  motion  even  though  the  inertial 
endpoints  were  intended.  The  PEST  procedure  was  used  to  systematically  alter  the  inertial 
amplitude  up  and  down  between  trials  to  find  the  inertial  amplitude  at  which  participants 
"crossed-ovef"  between  correctly  signaling  the  inertial  motion  and  falling  into  visual 
capture. 

The  two  sessions  were  usually  conducted  on  separate  days,  but  always  at  least  four  hours 
apart.  The  sessions  were  identical  except  that  the  order  of  the  first  condition  was  counter¬ 
balanced  across  sessions  and  in  one  of  the  sessions  an  Embedded  Figures  Test  (a  measure 
of  frame  dependency)  was  administered. 


Results: 

For  both  the  cross-over  and  reported  presence  data,  a  3-factor  ANOVA  was  computed  with 
factors  of  participants  (12),presence  manipulation  (meaningful/random)  and  session 
number  (first/second).  Three-way  and  higher  interactions  were  collapsed  into  the  error 
term. 

For  the  cross-over  measure,  main  effects  were  found  for  participants  (p  <  .01)  and 
treatment  (p  <  .01).  The  treatment  effect  was  in  the  direction  of  a  higher  inertial  cross-over 
for  the  meaningful  condition,  as  predicted.  A  participant-by-time  interaction  was  also 
found  (p  <  .01)  indicating  that  some  participants  had  a  lower  cross-over  on  the  second 
session.  No  other  effects  were  significant  at  p  <  .05. 

For  the  reported  presence  measure,  main  effects  were  found  for  participants  (p  <  .01)  and 
treatment  (p  <  .05).  The  treatment  effect  was  in  the  direction  of  a  higher  inertial  cross-over 
for  the  meaningful  condition,  as  predicted.  No  other  effects  were  significant  at  p  <  .05. 

The  test-retest  correlations  were  good  for  both  measures:  0.83  for  the  cross-over  measure, 
and  0.80  for  the  reported  presence  measure. 

The  mean  of  the  two  sessions  was  taken  for  all  participants  on  the  two  conditions.  The 
correlations  with  the  Embedded  Figures  Test  are  given  below. 

Cross-over  Reported  Presence 

Meaningful  .45  .04 

Random  .42  .28 

Within  subjects,  the  correlation  between  the  magnitude  of  cross-over  amplitudes  and  the 
magnitudes  of  reported  presence  was  .06. 

Within  subjects,  the  correlation  between  the  differences  in  the  cross-over  amplitude  across 
the  meaningful/random  eonditions,  and  the  differences  in  reported  presence  across  the 
meaningful/random  conditions,  was  0.38. 
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Discussion: 


Both  the  cross-over  and  reported  presence  measures  found  a  main  effect  for  condition  in 
the  predicted  direction  (meaningful  higher  than  random).  Both  measures  showed 
reasonable  test-retest  reliability  (0.83  and  0.80,  respectively). 

On  limited  data,  the  correlation  of  the  cross-over  measure  with  the  embedded  figures  test 
scores  appears  to  be  higher  than  the  correlation  of  the  reported  presence  measure  with  the 
embedded  figures  test  scores.  However,  this  result  has  to  be  interpreted  with  caution  for 
the  small  number  of  participants  run. 

It  is  not  surprising  that  no  relationship  (0.06  correlation)  was  found  between  the  magnitude 
of  cross-over  measures  and  the  magnitude  of  reported  presence.  This  is  consistent  with  the 
lack  of  a  standard  scale,  between  participants,  on  how  to  assign  numbers  to  mental  states. 
However,  there  is  a  weak  relationship  (0.38  correlation)  between  differences  in  conditions 
across  the  two  measures. 

Conclusion: 

This  preliminary  research  has  shown  that  the  cross-over  measure  is  capable  of  finding  a 
main  effect  in  the  predicted  direction  which  agrees  with  reported  presence  scores,  and  that 
the  test-retest  reliabilities  for  the  two  measures  are  similar.  It  finds  a  trend  for  a  stronger 
relationship  between  the  cross-over  measure  and  frame  dependency  (as  measured  by  the 
Embedded  Figures  Test)  than  between  reported  presence  and  frame  dependency. 

The  advantage  of  the  cross-over  measure  over  reported  presence  is  that  the  cross-over 
measure  is  based  on  the  trade-off  between  visual  and  inertial  motion  perception,  rather  than 
on  self-reported  numbers.  The  cross-over  measure  is  rooted  more  deeply  in  the  function  of 
the  nervous  system.  While  the  current  research  has  not  shown  this,  it  is  plausible  to 
suggest  that  a  multi-modal  nulling  measure  may  have  more  meaning  than  reported  presence 
between  participants  and  between  experiments. 

Thus,  between-subjects  differences  in  the  cross-over  ratings  may  relate  more  strongly  to 
performance  differences  than  is  true  for  reported  presence,  and  cross-over  amplitudes 
found  in  different  experiments  may  be  more  easily  comparable.  These  are  both  testable 
predictions. 
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2.2.  The  Virtual  Pilot  (ViP) 

The  problem 

The  problem  that  the  ViP  addresses  is  captured  by  the  phrase  “multi-crew  performance  in 
complex  military  systems.”  It  is  worth  considering  some  of  the  changes  to  the  mission  that 
have  precipitated  the  problem.  These  changes,  and  their  implications,  are  outlined  in  Table 
1. 


Changes 

Implications 

Smaller  military 

No  capacity  for  redundancy 

Current  or  increasing  tasking  levels 

Intelligent  implementation  of  pieces 

Synergy  between  the  pieces 

Larger  range 

Good  great  communications 

Dispersed  teams 

More  uncertainty 

Rapidly  forming  and  mobilizing  teams 

More  flexibility 

More  timely  response 

Less  time  for  training 

UAVs,  UCAVs 

Manned  missions  become  more  critical  and 
flexible 

Pilots  need  to  deal  with  more  information 

More  reliance  on  timely  information 

More  reliance  on  advisors 

Table  1.  Changes  in  the  mission,  and  the  implications  of  those  changes 


Basically,  there  is  an  increased  need  for  geographically  dispersed  teams  to  work  together  as 
cohesive  units.  These  teams  may  be  of  the  type  currently  implemented  (intelligence 
analysis,  fire  control,  air  traffic  control  etc.),  or  they  may  be  of  a  type  which  will  form  as  a 
result  of  the  changes  that  we  are  postulating. 

An  example  of  this  new  type  of  team  is  shown  Figure  3.  The  drawing  shows  a  shared 
virtual  environment  (VE)  occupied  by  four  people,  three  of  whom  are  represented  as 
avatars  (computer-generated,  human-controlled  people).  The  drawing  shows  the  scene 
from  the  real  physical  location  (an  AW  ACS)  of  the  human  in  the  center.  He  is  wearing  a 
see-through  head-mounted  display  which  allows  him  to  see  avatar  representations  of  the 
other  participants  in  virtual  space,  as  well  as  his  controls  and  displays  in  real  space.  The 
real  location  of  the  person  on  the  extreme  left  is  an  Air  Force  Base,  of  the  pilot  is  in  an 
airborne  F22,  and  of  the  person  at  the  bottom  of  the  drawing  is  in  the  Pentagon. 
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Figure  3.  An  example  of  geographically  dispersed  members  of  a  team  collaborating 
in  the  same  virtual  space. 


The  advent  of  UAVs  and  UCAVs  will  change  the  nature  of  manned  missions.  There  will 
always  be  manned  missions,  because  of  the  flexibility  and  performance  that  a  real  person 
can  bring  to  the  mission.  It  is  for  this  very  reason  that  those  missions  that  are  most 
sensitive,  complex  and  dynamic  will  remain  manned.  Therefore,  it  is  likely  that  the  pilots 
performing  those  missions  will  be  inundated  with  constantly  changing  information 
reflecting  changing  conditions.  They  will  benefit  from  advisors  -  virtual  backseaters  -  as 
depicted  in  the  diagram. 

There  is  another  problem  that  currently  exists.  This  is  captured  by  the  statement  made  by 
some  fighter  pilots  that  the  key  to  air-to-air  success  is  to  turn  of  the  AW  ACS  radio.  This 
could  be  because  the  perceived  value  of  the  AW  ACS  advisors  is  low,  or  because  the 
workload  of  the  pilots  is  too  high  to  deal  with  the  additional  information.  We  will  argue 
that  in  the  former  case,  the  problem  has  more  to  do  with  limited  communication  channels 
and  options  between  the  pilot  and  the  AW  ACS  advisor  rather  than  any  lack  of  ability  of  the 
advisor.  Also,  that  giving  the  advisor  more  information  about  the  pilot’s  situation  will 
increase  the  pilot’s  confidence  in  the  advice  being  given.  Furthermore,  during  periods  of 
high  workload  the  options  being  explored  in  the  ViP  concept  will  provide  ways  for  the  pilot 
to  offload  some  of  their  task  to  the  advisor. 
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The  solution  -  the  Virtual  Pilot 
Evolution  of  the  ViP  concept 

The  virtual  pilot  has  undergone  an  evolution  since  its  inception  in  September  1996.  Since 
then  there  have  been  three  ViPs.  These  are  described  below. 

ViP  (1),  September  1996 

In  our  first  Virtual  Pilot  demonstration,  two  people  sat  back  to  back  wearing  non-see- 
through  HMDs  (see  Figure  4).  One  played  the  role  of  pilot,  the  other  the  role  of  the  co¬ 
pilot.  Each  could  see  a  visual  representation  of  the  other  when  they  looked  to  the  side  (see 
Figure  5).  The  virtual  space  around  them  contained  moving  targets.  The  co-pilot  had  to 
identify  and  track  the  targets  so  that  the  pilot  could  shoot  them.  The  pilot  also  had  to 
perform  a  tracking  task.  Both  crew  members  had  to  collaborate  in  order  to  survive. 

The  lessons  learned  from  this  demonstration  were: 

1)  Context.  Some  of  the  gestural  communication  that  was  exchanged  between  the 
participants  alluded  to  context.  The  use  of  these  contextual  cues  seemed  to  make 
communication  easier. 

2)  Availability  for  communication.  An  important  benefit  of  being  able  to  see  the  other 
person,  or  at  least  an  avatar  representation,  and  what  they  were  doing,  was  that  it  became 
possible  to  determine  when  the  other  person  was  available  for  communication. 

3)  Augmented  reality.  The  demonstration  provided  information  that  would  not  have  been 
visible  in  the  real  world.  For  example,  where  the  co-pilot’s  hand  and  the  pilot’s  head  were 
pointing  in  space  was  visible  to  the  other  participant  (each  had  a  pointer  attached  to  them). 
This  disambiguated  the  contextual  information  and  made  communication  easier. 

4)  Display  design.  Forcing  the  pilot  to  look  at  the  co-pilot  was  not  the  most  efficient  way 
of  designing  an  alerting  display.  It  would  have  been  better  to  have  a  head-mounted 
designator  symbol  showing  the  pilot  where  to  look. 

With  these  lessons  in  mind  we  designed  the  second  ViP  demonstration. 
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Figure  4.  Plan  view  of  the  layout  used  in  ViP  1. 


Figure  5.  View  of  the  pilot  (right  seat)  and  co-pilot  (left  seat)  in  the  ViP  1 
demonstration. 


ViP  (2),  May  1997 

In  the  second  of  our  virtual  pilot  demonstrations  we  changed  the  emphasis  from  the  pilot 
viewing  the  advisor  to  the  advisor  viewing  the  pilot.  The  pilot’s  task  was  to  land  on  an 
aircraft  carrier.  The  advisor  viewed  the  pilot’s  avatar.  By  giving  the  advisor  a  view  of  the 
pilot’s  instruments,  and  their  head  position,  we  intended  for  the  advisor  to  get  a  better 
understanding  of  what  the  pilot  was  thinking  (a  view  inside  their  mind).  This  line  of 
thinking  resulted  in  the  concept  of  “telesavance”. 

Telesavance  is  the  communication  of  the  state  of  someone’s  situation  awareness. 
“Savance”  comes  from  the  term  “savant”  -  a  wise  or  knowledgeable  person.  Information 
about  the  pilot’s  SA  is  important.  It  allows  the  advisor  to  know  what  the  pilot  knows  and 
doesn’t  know.  In  face-to-face  encounters  telesavance  is  communicated  by  the  methods 
listed  in  Table  2.  This  certainly  includes  auditory  and  visual  channels,  and  may  include 
others.  Some  of  the  information  may  be  used  consciously  and  some  may  be  used 
unconsciously  (e.g.  pupil  dilation).  Another  important  element  of  telesavance  is  the  mental 
model  that  the  advisor  has  which  allows  him  to  predict  how  the  pilot  may  react.  The 
combination  of  the  communication  channels  and  options,  and  the  mental  model,  allows  the 
advisor  to  be  a  better  advisor.  In  contrast  to  the  channels  and  options  available  in  face-to- 
face  communication,  the  current  methods  of  communicating  telesavance  in  aircraft  are  also 
shown  in  Table  2. 


Figure  6.  View  of  the  pilot’s  avatar  from  the  point  of  view  of  the  advisor.  The  line 
extending  from  the  pilot’s  head  shows  where  his  head  is  pointing.  This 
was  the  arrangement  in  ViP  2. 
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Channels  and  options 

Face-to-face 

Verbal 

V 

Gestural 

V 

Facial  expression 

V 

Shared  information  displays 

V 

Responses  to  verbal 

V 

Responses  to  gesture 

V 

Performance 

V  . 

Eye  movement 

V 

Pupil  dilation 

V 

Other  physiological  information 

V 

blushing 

v 

sweating 

V 

breathing 

V 

trembling 

V 

swallowing 

V 

tearing 

V 

odor 

V 

In  aircraft 


Table  2.  How  telesavance  is  communicated  face-to-face  and  in  aircraft. 


ViP  (3),  October  1997 

In  our  third  ViP  demonstration  we  measured  the  pilot’s  eye  motion  and  allowed  the  advisor 
to  view  it  via  a  see-through  head-mounted  display.  The  advisor  saw  a  reticle  with  a 
decaying  trail  that  corresponded  to  where  the  pilot  was  currently  looking  and  had  most 
recently  looked.  This  was  a  demonstration  of  augmented  reality.  For  this  demonstration 
we  used  a  commercially  available  flight  simulator,  and  blanked  out  portions  of  the  pilot’s 
and  advisor’s  screens  to  simulate  the  disparity  in  information  between  a  pilot  and  an 
AW  ACS  operator. 
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Figure  7.  The  pilot  and  advisor  as  arranged  in  ViP  3.  The  pilot  saw  a  subset  of  the  flight  simulator 
information  on  the  projection  screen.  The  advisor  saw  a  different  subset  on  his 
monitor,  but  he  could  also  see  what  the  pilot  saw  by  looking  at  the  projection  screen. 
Because  the  advisor  wore  a  see-through  HMD,  he  could  also  see  the  pilot’s  eye  motion 
superimposed  on  the  projection  screen  (augmented  reality). 


Figure  8.  Component  diagram  from  ViP  3.  The  combination  of  the  pilot’s  eye 

and  head  motion  (gaze)  and  the  advisor’s  head  motion  were  used 
to  create  a  reticle  that  followed  the  pilot’s  gaze,  and  was  visible 
through  the  advisor’s  HMD. 
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An  experimental  test  of  the  concept 

We  have  conducted  an  experiment  to  test  the  utility  of  the  ViP  concept.  The  experiment 
drew  on  elements  refined  from  all  of  the  ViP  demonstrations.  It  was  the  first  attempt  at  an 
empirical  validation,  and  as  such  used  a  simple  paradigm,  a  simple  experimental  design, 
and  simple  apparatus.  We  reasoned  that  we  needed  to  determine  whether  or  not  there  was 
an  effect  (i.e.  an  advantage  of  face-to-face  communication  over  verbal-only 
communication)  before  we  embarked  on  explorations  of  the  nature  and  causes  of  that 
effect. 

Our  method  of  testing  the  hypothesis  was  to  compare  a  low-tech  method  of 
communication,  namely  voice-only,  with  what  we  termed  a  high-tech  communication 
method,  and  to  measure  mission  performance,  situation  awareness  of  the  pilot  and  advisor, 
and  telesavance. 

Apparatus 

Our  high-tech  communication  method  was  in  fact  a  simple  simulation  of  what  could  be 
achieved  if  we  had  equipment  sophisticated  enough  to  put  two  geographically  dispersed 
members  of  a  team  in  the  same  virtual  space,  with  sufficient  fidelity  for  them  to  think  that 
they  were  sitting  in  the  same  physical  space.  We  did  this  by  actually  putting  them  in  the 
same  physical  space  -  thereby  using  physical  proximity  to  simulate  high-tech 
communication  methods. 

The  experiment  simulated  a  team  of  a  fighter  pilot  and  an  AW  ACS  operator.  In  order  to  do 
this  we  used  a  commercially  available  flight  simulator  (iF  22  from  Interactive  Magic).  We 
used  a  video  splitter  to  show  the  same  seene  on  two  CRT  screens,  and  selectively  blanked 
areas  of  both  screens  to  simulate  the  disparity  of  information  between  a  pilot  and  an 
AW  ACS  operator.  The  information  on  each  screen  is  shown  in  Figure  9.  The  pilot’s  view 
was  essentially  ego-centric,  whereas  the  advisor’s  view  was  exo-centric.  Figure  10  shows 
the  arrangement  of  the  two  participants.  In  the  high-tech  condition  the  advisor  could  see 
both  his  own  screen  and  the  pilot’s  screen,  and  therefore  increase  his  SA  substantially. 

The  pilot’s  view  was  the  same  in  both  conditions. 

The  choice  of  what  flight  simulator  to  use  was  made  after  an  analysis  of  the  available 
products.  Over  the  past  10  years  the  advent  of  high  performance  PCs  and  graphics  cards, 
and  the  market  pull  of  the  $16b  computer  games  market  have  created  flight  simulators  with 
a  level  of  realism  that  matches  the  best  military  simulators.  The  game  we  decided  on  has  a 
variety  of  missions,  enemy  skill  levels  and  fighter  performances.  It  is  capable  of  being 
networked  for  head-to-head  dogfights  or  group  missions.  The  computer  opponents  and 
wingmen  incorporate  artificial  intelligence,  so  that  no  two  missions  are  exactly  the  same. 
The  visual  scenery  is  based  on  satellite  imagery.  Missions  can  be  individually  tailored  and 
then  saved  and  re-run  .  All  this  for  $49.95. 
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Figure  9.  Information  available  on  the  pilot’s  and  advisor’s  screens. 
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VOICE-ONLY 


HI  TECH 


Figure  10.  Arrangement  of  the  pilot  and  advisor  in  the  two  experimental 
conditions. 

Dependent  variables 

The  dependent  variables  were  chosen  to  measure  performance,  situation  awareness  (SA) 
and  telesavance.  For  performance  we  used  the  simulators  statistics  that  were  calculated  and 
made  available  at  the  end  of  each  mission.  These  were  Battle  Damage  Assessment  (BDA,  0 
-  100%),  Result  (poor,  average,  outstanding),  effectiveness  (0  -  100%),  number  of  kills, 
mission  duration,  and  number  of  crashes.  SA  was  measured  for  both  pilot  and  advisor 
using  a  modified  Situation  Awareness  Global  Assessment  Technique  (SAGAT)  after 
Endsley  (1995).  It  is  unusual  to  measure  SA  for  all  members  of  a  team.  Very  little 
research  has  focused  on  the  team  element  in  SA  (Salas  et  al,  1995).  Telesavance  was 
measured  indirectly  using  a  series  of  visual  analogue  scales.  Both  the  pilot  and  the  advisor 
were  asked  to  rate  the  amount  of  communication,  the  value  of  the  communication,  the 
instigator  of  communication,  the  advisor’s  performance  and  the  pilot’s  performance.  In 
addition,  each  trial  was  videotaped  for  later  transaction  analysis  and  objective  measures  of 
communication. 

Experimental  design 

The  subjects  used  in  the  data  reported  here  were  1 1  civilian  users  of  the  game,  and  an  ex 
Navy  pilot  instructor  with  over  2000  hours  in  fast  jets,  including  carrier  landings.  The 
advisor  was  an  expert  civilian  user. 
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The  order  of  presentation  of  the  two  communication  conditions  was  balanced  across 
subjects.  The  sequence  in  which  each  subject  was  run  was  as  follows; 

Train  to  criterion  -  each  subject  was  trained  by  the  advisor  and  tested.  Before  being  able  to 
participate  they  had  to  be  able  to  fly  in  an  “Instant  Action”  scenario  and  achieve  two  air-to- 
air  kills  without  being  shot  down.  They  had  to  do  so  once  out  of  three  trials. 

Mission  1  -  the  subject  embarked  on  their  first  mission,  either  in  the  hi-tech  or  voice-only 
condition.  They  were  taken  at  8  times  normal  speed  to  the  edge  of  the  battle  area,  which 
was  a  Combat  Air  Patrol  over  Bosnia.  They  were  the  squadron  leader  of  a  4-man 
formation  with  3  other  F22s.  At  the  designated  waypoint  the  simulator  was  returned  to 
normal  speed  and  paused.  A  timer  was  started  and  the  mission  began.  Three  minutes  later, 
and  at  subsequent  3  minute  intervals  during  the  mission  the  simulator  was  paused  so  that 
both  the  pilot  and  advisor  could  fill  in  the  SAGAT  questionnaire  and  the  visual  analogue 
scales.  These  were  then  graded  before  the  mission  was  recommenced.  The  questionnaire 
was  administered  a  maximum  of  four  times  in  each  mission,  less  if  the  person  crashed. 

Mission  2  -  the  subject  embarked  on  the  second  mission,  in  the  opposite  condition  to  the 
first  mission  (i.e.  voice-only  after  the  high-tech  condition,  and  vice  versa).  All  other  events 
were  the  same  as  in  mission  1 . 

Results 

Figure  10  shows  mixed  evidence  of  better  performance  in  the  hi  tech  condition.  However 
the  variability  in  the  data  mean  that  these  differences  were  not  significant.  A  closer 
examination  of  the  data  was  conducted,  by  dividing  the  data  into  those  from  low  experience 
pilots  (n=5),  and  high  experience  pilots  (n=6).  Experience  was  assessed  from  a  screening 
questionnaire.  Figure  1 1  shows  the  results  from  the  low  experience  pilots.  There  is 
evidence  that  their  performance,  as  measured  by  most  of  the  indicators,  was  worse  in  the  hi 
tech  condition.  In  contrast.  Figure  12  shows  the  data  from  the  more  experienced  pilots,  in 
which  there  is  evidence  that  performance  improved  in  the  hi  tech  condition.  The  data  in 
Figure  13  provides  some  indication  as  to  the  cause  of  this  difference.  It  would  appear  that 
in  the  hi  tech  condition  the  less  experienced  pilots  were  overwhelmed  and  their  SA 
dropped.  As  a  result,  all  of  these  pilots  were  shot  down  (Figure  1 1).  The  more 
experienced  pilots,  on  the  other  hand,  benefited  from  the  increased  information  and  their 
performance  improved. 
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Figure  10.  Mean  performance  data  from  11  subjects 
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Figure  11.  Performance  data  from  the  low  experience  pilots  (n=5) 
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Discussion  of  the  results 

The  data  suggest  that  the  more  experienced  pilots  were  aided  by  the  technology,  resulting  in 
higher  SA  and  better  performance.  Less  experienced  pilots  were  overwhelmed  by  the 
added  information  and  showed  a  performance  decrement.  This  type  of  effect  is  consistent 
with  the  psychological  literature.  Experts  process  information  more  effectively  than 
novices  (by  chunking  etc.). 

In  the  operational  environment  the  people  using  the  technology  will  be  experts.  Therefore, 
as  a  test  of  the  utility  of  the  techniques,  the  previous  experiment  needs  to  be  extended  to 
include  experienced  milita^  pilots.  This  will  be  done  in  work  currently,  supported  by  the 
Armstrong  Laboratory,  using  pilots  from  Boeing.  The  HIT  Lab  has  forged  a  relationship 
with  Boeing  in  the  performance  of  this  work,  described  in  Section  5. 
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2.3.  Performance  study  of  the  Virtual  Motion  Controller 

Abstract  of  a  paper  proposal  to  the  Human  Factors  and  Ergonomics  Society  by  Peterson, 
Wells,  Furness  and  Hunt. 

Thirty  subjects  explored  mazes  in  an  immersive  virtual  environment.  Half  of  the  subjects 
used  a  joystick  for  maneuvering;  the  other  half  used  a  new  type  of  body-controlled  interface 
called  the  virtual  motion  controller  (VMC).  Maneuvering  performance,  as  measured  by  the 
precision  with  which  subjects  followed  a  marked  route,  was  slightly  worse  with  the  VMC 
than  with  the  joystick.  Route  learning,  as  measured  by  the  subjects’  ability  to  replicate  the 
route,  was  the  same  for  both  devices.  Survey  knowledge,  or  the  ability  to  form  a  mental 
map  of  the  space,  and  to  use  that  to  find  alternative  routes,  was  significantly  better  with  the 
VMC  than  with  the  joystick.  This  enhancement  in  performance  was  conditional  on  maze 
difficulty.  The  more  difficult  the  maze  the  greater  was  the  performance  benefits  of  using 
the  VMC.  The  experiment  provided  evidence  that  the  VMC  can  enhance  certain 
components  of  navigation  in  virtual  environments. 

Introduction 

When  we  navigate  in  the  real  world  we  have  several  types  of  sensory  cues  to  tell  us  where 
we  are  and  where  we  have  been.  These  include  cues  from  vision,  the  vestibular  apparatus, 
the  haptic  and  proprioceptive  senses,  and  audition.  Together  these  allow  us  to  acquire 
knowledge  about,  and  form  a  mental  representation  of,  a  space.  Researchers  have  broken 
down  this  knowledge  into  route  and  survey  knowledge  (Thomdyke  and  Hayes-Roth, 

1982).  Route  knowledge,  also  known  as  procedural  knowledge,  consists  of  the 
procedures  required  to  find  a  set  of  targets  in  the  world.  Survey  knowledge,  also  known 
as  configurational  knowledge  consists  of  a  more  flexible  “big  picture”  of  the  space  that  can 
be  used,  for  example,  to  find  alternative  routes. 

There  is  evidence  that  the  human  sensory  apparatus  is  tuned  to  integrate  information  from 
several  sensory  modalities  to  create  a  single  coherent  sensation  (Welch  and  Warren,  1986). 
When  environmental  information  is  limited  to  only  one  or  two  sensory  modalities, 
interpretations  of  that  environment  can  be  misleading  or  inaccurate,  thereby  increasing  the 
complexity  of  otherwise  simple  tasks  (Sherrick  and  Cholewiak,  1986).  Therefore,  when 
computer-mediated  interactions  strip  away  information  they  do  not  provide  the  complete 
and  expected  information  (even  if  this  information  may  be  “redundant”)  and  may  handicap 
performance  (Durlach  and  Mavor,  1995). 

Several  researchers  have  attempted  to  explore  navigation  in  virtual  environments  (e.g.  Bliss 
et  al,  1997,  Regian  et  al,  1992,  Ruddle  et  al,  1996,  Satalich,  1995,  Tlauka  and  Wilson, 
1994,  Witmer  et  al,  1996).  Indeed,  navigational  training  may  be  one  of  the  “killer  apps”  of 
immersive  VR.  However,  in  most,  if  not  all  cases  to  date,  the  interfaces  used  stripped 
away  many  of  the  sensory  stimuli  and  the  research  was  conducted  with  relatively  sparse 
sensory  interaction. 

In  an  attempt  to  put  back  some  of  the  multi-sensory  information,  we  created  a  new  interface 
device,  called  the  virtual  motion  controller  or  VMC  (Wells,  Peterson  and  Aten,  1996).  The 
VMC  uses  the  body  to  generate  motion  commands  and  provides  some  vestibular,  haptic 
and  proprioceptive  cues.  The  research  in  this  paper  describes  an  experiment  to  compare  the 
VMC  with  a  more  commonly  used  interface,  in  which  mainly  visual  cues  are  provided. 

Our  rationale  was  to  first  determine  whether  there  was  a  difference,  and  if  there  was,  to 
conduct  further  research  to  find  out  why.  This  is  the  first  step  in  that  experimental 
program. 
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Methods 

Apparatus 

A  diagram  of  the  apparatus  for  both  controller  conditions  is  shown  in  Figure  14.  The  head 

mounted  display  (HMD)  was  a  Vi/0 1  Glasses  ™  with  a  fully  overlapped  field  of  view  of 
32  degrees  horizontal  by  24  degrees  vertical.  The  display  in  each  eye  consisted  of  an  LCD 
with  320  pixels  by  200  pixels.  Subjects  could  see  around  the  edges  of  the  HMD,  but  an 
opaque  visor  blocked  their  view  through  the  display  area.  Two  computers  controlled  and 
generated  the  virtual  world.  Monocular  images  for  display  on  the  HMD  were  generated  by 
a  Silicon  Graphics  Onyx.  The  Onyx  maintained  a  frame  rate  in  excess  of  20  frarnes  per 
second.  The  input  devices  were  a  Polhemus  Fast  Track  for  measuring  head  motion,  the 
virtual  motion  controller  (VMC),  or  a  joystick.  The  VMC  consisted  of  a  disc  on  which  the 
subject  stood  (see  Figure  15).  It  worked  as  a  first  order  controller,  turning  displacements 
on  the  disc  into  velocities  in  the  VE.  Movements  away  from  the  center  of  the  VMC  resulted 
in  motion  in  the  virtual  environment  (VE)  in  the  same  direction  as  the  movement  on  the 
VMC,  and  with  a  velocity  proportional  to  the  distance  from  the  center.  The  joystick  was  a 
Sidewinder  Pro,  made  by  Microsoft.  Forward/backward  and  left/right  motion  of  the 
joystick  caused  forward/backward  translational  and  left/right  rotational  velocity  in  the  VE, 
proportional  to  the  amount  of  joystick  movement.  In  both  conditions,  yaw  axis  rotational 
displacement  of  the  head  caused  yaw  axis  rotational  displacement  in  the  VE. 

Virtual  Environments 

The  virtual  environments  consisted  of  three  mazes  of  vatying  complexity  -  a  simple  training 
maze,  a  simple  experimental  maze,  and  a  complex  experimental  maze  The  mazes  were 
designed  so  that  the  paths  through  them  were  non  orthogonal.  A  plan  and  subject’s  eye 
view  of  the  mazes  are  shown  in  Figures  16  and  17.  The  mazes  extended  over  an  area  of 
approximately  250  by  250  units.  A  fog  effect  reduced  visibility  in  the  maze  to  50  units.  In 
some  conditions  path  markers  showed  the  route  to  be  learned.  Collision  detection  was 
implemented  such  that  subjects  could  pass  through  the  interior  walls,  but  not  the  exterior 
walls  of  the  maze.  Maze  difficulty  was  varied  by  manipulating  the  total  path  distance  and 
degrees  of  turning. 

Design  and  procedure 

The  independent  variables  were  controller  type  (VMC  and  joystick  (contracted  to  SW  for 
Sidewinder)),  and  maze  difficulty.  The  controller  type  was  tested  between  subjects,  and 
maze  difficulty  was  tested  within  subjects.  Table  3  shows  the  order  in  which  the 
experiment  was  conducted.  The  training,  easy  maze,  and  difficult  maze  conditions  were 
tested  with  the  same  protocol  (described  below),  with  the  exception  that  in  the  training 
condition  subjects  were  encouraged  to  ask  questions  and  any  problems  were  sorted  out. 
Each  maze  condition  consisted  of  a  learning  phase  of  5  trials  with  the  path  markers,  during 
which  the  subject  tried  to  learn  the  route.  After  each  trial  the  subject  was  transported  back 
to  the  start  of  the  maze  and  asked  to  point  to  the  exit  and  report  their  pointing  confidence, 
and  their  confidence  of  route  replication  without  markers.  After  5  learning  trials  the  path 
markers  were  removed  and  the  subject  then  had  2  trials  in  which  they  tried  to  replicate  the 
route.  After  2  replication  trials  the  subject  did  2  trials  in  which  they  tried  take  the  shortest 
straight  line  route  to  the  exit,  passing  through  any  interior  walls  to  do  so.  Before  the 
training  session  subjects  read  instructions  and  completed  a  Sickness  Symptom 
Questionnaire  (SSQ)  (Kennedy  et  al  1993).  Subjects  were  given  a  short  break  between  the 
training,  easy,  and  difficult  mazes,  and  after  the  difficult  maze  they  were  given  another 
SSQ  and  a  session  feedback  form. 
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We  decided  against  counter  balancing  maze  difficulty  because  of  the  possibility  of 
asymmetric  transfer  (e.g.  a  different  strategy  being  used  and  transferred  when  the  easy 
maze  was  presented  first  than  when  the  difficult  maze  was  presented  first). 

Dependent  variables 

The  dependent  variables  reported  here  are:  MARKERS  HIT  (subjects  were  instructed  to 
pass  over  or  “hit”  all  of  the  route  markers  in  the  learning  phase),  DISTANCE  PER 
MARKER  (distance  traveled  as  the  subject  went  from  route  marker  to  route  marker  during 
the  learning  phase  -  a  measure  of  the  precision  of  maneuvering),  PERCENT  LOST  (the 
percent  of  participants  who  got  lost  during  the  straight  line  phase,  as  determined  by  their 
paths  making  two  or  more  loops,  and/or  by  them  making  two  or  more  incorrect  course 
corrections),  DISTANCE  (distance  traveled  from  the  entrance  to  the  exit  during  the  straight 
line  phase,  for  the  trials  in  which  the  subjects  were  not  lost),  ESTIMATED  ANGLE 
(subjects  used  a  head-fixed  reticle  to  point  to  the  exit  from  the  entrance  at  the  end  of  each 
learning  trial).  The  location  of  the  subject  was  sampled  and  recorded  every  second.  A 
number  of  other  dependent  variables  were  derived  from  this  information,  but  are  not 
reported  here. 

Subjects 

Thirty  people  (12  females  and  18  males)  participated  in  the  experiment.  Most  were  staff 
and  students  at  the  University  of  Washington.  Their  ages  ranged  from  18  to  50. 

Results 

The  results  can  be  considered  to  consist  of  maneuvering  performance,  route  replication 
performance  or  route  knowledge,  and  survey  knowledge.  Route  knowledge,  as  measured 
by  the  confidence  with  which  subjects  felt  they  could  replicate  the  route,  and  their 
performance  at  doing  so,  was  not  significantly  different  across  the  devices.  These  data  will 
not  be  presented  here. 

Maneuvering  performance  was  slightly  worse  with  the  VMC.  The  number  of  markers  hit, 
shown  in  Figure  18,  indicates  that  with  the  VMC  the  participants  hit  slightly  less  markers 
than  with  the  joystick  (Mann-Whitney  U-Test  U=73.5,  p=0.10).  This  trend  is  consistent 
with  the  distance  per  marker,  shown  in  Figure  19,  which  indicates  a  small,  but  significant 
benefit  of  using  the  joystick  (U  =  1.0,  p<0.05).  The  explanation  is  that  (l)the  VMC 
required  more  motion  for  control  actions  than  the  joystick  (2)  unlike  the  joystick,  the  VMC 
turned  comers  in  a  series  of  straight  lines,  resulting  in  less  precise  path  control  (3)  the 
VMC  was  a  research  device,  and  was  not  as  optimized  for  control  as  the  commercial 
joystick  and  (4)  most  subjects  had  more  experience  with  joysticks  than  with  the  VMC. 

Survey  knowledge,  as  measured  by  how  well  subjects  could  find,  or  point  to,  a  straight 
line  path  to  the  exit,  was  much  better  with  the  VMC.  Figure  20  shows  the  percentage  of 
people  who  became  lost  when  asked  to  go  straight  to  the  exit.  Participants  using  the 
joystick  were  more  than  3  times  as  likely  to  get  lost  as  participants  using  the  VMC  (18.3% 
vs  5. 1  %).  The  difference  between  devices  was  not  significant  with  the  simple  maze 
(Z=1.23,  p  =  0.10),  but  was  significant  for  the  complex  maze  (Z=1.98,  p  <0.05).  The 
shortest  straight  line  paths  for  the  simple  and  complex  mazes  were  207  and  209  units 
respectively.  Figure  21  shows  that  the  mean  distance  for  the  simple  maze  was  just  under 
300  units,  and  that  the  joystick  and  VMC  performed  about  equally  well.  However  with  the 
complex  maze  the  VMC  significantly  outperformed  the  joystick.  A  mixed  factor  ANOVA 
shows  that  the  interaction  was  significant  (1,28  F=4.84  p<0.05).  Finally,  the  pointing 
angle  data  in  Figure  22  indicate  how  well  the  subjects  were  able  to  assess  the  direction  of 
the  exit  from  the  entrance  during  all  the  trials  in  the  route  learning  phase.  The  VMC  group 
had  significantly  smaller  pointing  errors  (1,28  F=  13.20  p<0.01).  Pointing  was  more 
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accurate  in  the  simple  maze  (1,  28  F=22.54  p<0.01).  The  maze  x  device  interaction  was 
also  significant  (1,28  F=4.85  p  <0.05). 

Discussion 

The  results  provide  evidence  that  the  VMC  enhances  certain  aspects  of  navigation 
performance  in  virtual  environments.  Specifically,  users  were  able  to  create  a  more 
accurate  mental  map  of  the  space  with  the  VMC  than  with  a  joystick.  The  enhancement 
was  dependent  on  the  complexity  of  the  maze,  with  more  complex  mazes  showing  more  of 
a  benefit  for  the  VMC.  The  results  point  to  an  interesting  line  of  research  to  determine 
why. 
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Figure  14  Diagram  of  the  apparatus  used  in  the  experiment. 
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Table  3  Design  of  the  experiment.  The  type  of  controller  (VMC  -  virtual  motion 
controller,.  SW  -  Sidewinder  joystick)  was  tested  between  subjects.  Maze 
difficulty  (easy,  difficult)  was  tested  within  subjects. 
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Figure  15  Photograph  of  the  VMC  and  of  a  subject  kneeling  on  the  device. 
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Figure  1 6.  Plans  of  the  simple  (top)  and  complex  (bottom)  mazes. 
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Figure  17.  Immersive  view  of  one  of  the  mazes 
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Figure  18.  Average  number  of  markers  hit  (more  =  better).  There  were  14  and  15 
markers  in  the  simple  and  complex  mazes  respectively. 
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Figure  19.  The  distances  traveled  between  each  route  marker  (more  =  worse). 


Figure  20.  Percentage  of  subjects  who  got  lost  while  trying  to  find  a  straight  line 
route  from  the  entrance  to  the  exit. 
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Figure  21 .  The  distances  traveled  while  finding  a  straight  line  route  from  the 

entrance  to  the  exit  (optimal  distances  were  207  and  209  units  for  the  simple 
and  complex  mazes).  The  data  are  from  the  subjects  who  were  NOT  lost. 


Figure  22.  The  estimated  angular  error  when  subject’s  were  asked  to  point  to  the 
exit  from  the  entrance. 
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2.4.  The  Go-Go  Interaction  Technique 

As  most  users  of  virtual  reality  systems  have  discovered,  grasping  virtual  objects  that  are 
out  of  reach  can  be  challenging,  especially  given  the  often  inadequate  distance  cues 
currently  available.  Users  often  find  themselves  flying  backwards  and  forwards  in  a  vain 
attempt  to  position  an  object  within  reach.  This  can  lead  to  user  confusion  and  frustration. 

In  fact,  a  report  by  the  National  Research  Council’s  Committee  on  Virtual  Reality  Research 
and  Development  [1]  identified  the  problem  of  selecting  and  manipulating  objects  located 
outside  of  the  user’s  reach  as  one  of  the  most  challenging  problems  in  immersive  interface 
development. 

The  Go-Go  interaction  technique  allows  immersed  participants  to  select  and  manipulate  any 
virtual  object  in  sight  without  having  to  reposition  their  virtual  body.  The  direct 
manipulation  technique  is  based  on  the  metaphor  of  being  able  to  change  arm  length  at  will 
(Figure  1),  and  was  named  the  “Go-Go  interaction  technique,”  after  the  cartoon  character 
Go-Go  Gadget,  who  had  the  unique  ability  to  telescope  his  arms  beyond  normal  reach. 

The  Go-Go  metaphor  is  implemented  by  using  a  nonlinear  function  for  mapping  the 
movement  of  the  user’s  physical  hand  to  the  affected  movement  of  the  virtual  hand  (Figure 
2).  To  reach  and  manipulate  distant  objects,  the  user  extends  a  hand  toward  the  object  of 
interest  and,  because  the  mapping  uses  a  nonlinear  positioning  function,  the  user’s  area  of 
reach  is  expanded  beyond  normal.  To  allow  for  nearby  manipulation,  a  step  function  is 
used:  the  mappmg  is  linear  in  the  space  close  to  the  user  and  accelerates  smoothly  as  the 
user  extends  her  physical  arm  outside  of  the  envelope  of  linear  manipulation. 

Preliminary  user  evaluations  indicate  the  Go-Go  interaction  technique  is  extremely  intuitive, 
as  it  imitates  real  world  human  behavior:  when  we  want  to  reach  a  remote  object  we  stretch 
an  arm  toward  :t.  It  is  thus  the  only  technique  which  allows  seamless  direct  manipulation  of 
objects  both  near  and  far. 

[1]  Durlach  N.  and  Mavor  A.,  Eds.  “Virtual  Reality:  Scientific  and  I'echnological 
Challenges,”  National  Academy  Press,  1995,  p.  542. 
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5. 


Ongoing  and  Future  Research 


One  of  the  goals  of  the  AFOSR  MURI  program  was  to  initiate  research  at  HITL.  This  was 
successfully  accomplished.  Some  of  the  research  thrusts  have  taken  on  a  life  of  their  own, 
with  some  evolutionary  changes.  Some  current  and  proposed  research  efforts  with  direct 
links  to  this  grant  are  described  below. 

5.1.  Computer-mediated  communication 

5. 1.1.  The  SHARE  Consortium 

In  December  1996  Boeing  created  a  consortium  of  participants  who  can  contribute  to  the 
building  of  the  SHARE  vision.  These  include  Microvision  -  the  makers  of  the  Virtual 
Retinal  Display,  GEC  -  a  helmet-mounted  avionics  manufacturer,  Boeing  -  as  the  prime 
integrator,  and  the  HIT  Lab  -  for  concept  development  and  as  the  research  and  development 
arm.  The  original  consortium  submitted  a  proposal  to  DARPA  under  the  Warfighter 
Visualization  program.  This  proposal  was  not  funded,  but  the  consortium  lived  on  and 
continued  to  incubate  the  concept  and  generate  components  with  their  own  resources.  In 
December  of  1997  a  briefing  was  given  to  the  Army  and  Navy  at  a  meeting  at  Boeing. 
Prospective  additions  to  the  consortium  include  Centex  -  helmet  and  helmet-mounted 
display  integrators,  and  Applied  Sciences  Laboratory  -  eye  tracker  manufacturers. 

The  consortium  represents  an  amalgamation  of  industrial  and  academic  entities  with  a  clear 
vision,  momentum,  and  a  commitment  to  the  successful  integration  of  technology  to 
enhance  military  performance. 

Future  experiments 

We  are  designing  a  series  of  experiments  which  will  provide  a  better  understanding  of  the 
issues  being  raised  by  the  current  work,  and  which  will  address  questions  relevant  to  the 
future  implementation  of  the  ViP  concept.  These  include: 

•  The  effects  of  transactive  memory  (using  the  same  vs.  different  advisors) 

•  Advisor  and  pilot  skill  level 

•  The  contributions  of  the  various  channels  of  communication 

•  The  most  appropriate  technological  implementation  (e.g.  VR  vs.  video) 

•  The  effects  of  workload 

•  The  role  of  eye  movement  (and  other  technology  enhanced  communication 
options) 

5. 1.2.  Knowledge  and  Distributed  Intelligence 

We  have  responded  to  an  NSF  call  for  proposals,  using  the  ViP  work  as  a  base.  Below  is 
our  letter  of  intent. 

Project  Title:  Knowledge  Profiler 
KDI  Focus:  Knowledge  Networking 

Project  Description: 

In  the  paradigm  of  knowledge  networking  mediated  by  computers,  there  remain  three  key 
interactions  which  can  impact  the  way  in  which  knowledge  is  created,  communicated  and 
valued.  These  are:  (1)  human-to-computer  interactions,  (2)  computer-to-human 
interactions  and  (3)  human-to-human  interactions.  We  believe  that  improvements  in  any 
one  can  create  substantial  payoffs,  but  that  1  and  2  are  high  risk  (we  are  a  long  way  from 
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understanding  the  fundamental  components  of  real  or  artificial  intelligence  that  would  be 
required  for  peer-to-peer  communication).  The  risk  can  be  reduced,  while  maintaining 
most  of  the  benefit,  by  creating  a  hybrid  system  in  which  a  human  interprets  the  output 
from  a  computer  and  communicates  that  to  another  human.  In  essence  this  is  a  fourth 
option,  termed  computer-to-human-to-human  interaction.  The  benefits  of  this  sort  of 
interaction  occur,  for  example,  between  an  advisor  and  a  person  being  advised,  where  the 
person  receiving  advice  cannot  devote  the  attentional  resources  required  to  deal  with  an 
unintelligent  machine.  Apart  from  improving  performance,  this  fourth  option  also  p»rovides 
a  rich  testbed  for  understanding  the  nature  of  the  communication  of  knowledge.  This 
understanding  could,  in  due  course,  be  used  increase  machine  intelligence. 

One  factor  that  we  have  identified  as  being  important  in  this  type  of  knowledge  transfer  is 
that  the  advisor  be  aware  of  the  state  of  knowledge  of  the  person  being  advised.  We 
propose  the  creation  of  a  Knowledge  Profiler,  which  uses  a  combination  of  display  design, 
eye  movement  tracking,  and  other  behaviors  to  provide  information  to  the  advisor  about 
what  advice  is  needed. 

The  project  will  match  a  technology  push,  as  represented  by  the  lines  of  research  being 
pursued  by  some  members  of  the  team,  with  a  requirements  pull,  as  represented  by  the 
need  for  collaborative  innovation  and  knowledge  sharing  brought  by  the  industrial  partners. 
The  research  to  be  leveraged  includes  (1)  the  SHARE  project,  in  which  members  of  a  team 
use  shared  augmented  reality  to  vanquish  the  limitations  of  time  and  distance  (2) 

TADMUS,  which  uses  decision  theory  and  human-system  interaction  technology  in  a 
decision  support  system  and  (3)  the  Virtual  Pilot  project,  which  uses  eye  tracking  as  a  way 
to  communicate  situation  awareness  between  a  pilot  and  an  advisor. 

5.2.  Motion  sickness  and  the  Rest  Frame  Hypothesis 

5.2.1. VMC 

We  have  continued  to  work  on  the  VMC.  Our  most  recent  experiment  seeks  to  explain 
why  the  VMC  is  better  at  creating  survey  knowledge.  Our  first  exploration  manipulated 
whether  or  not  subjects  received  “functional  distal  cues”.  In  other  words,  we  manipulate 
whether  subjects  could  see  around  the  edges  of  the  head-mounted  display  to  view  the 
laboratory.  Our  hypothesis  is  that  the  subjects  use  the  stationary  landmarks  in  the  lab  as 
landmarks  in  the  virtual  world.  We  used  blinders  which  blocked  the  subjects  view  of 
anything  other  than  the  content  of  interest  on  the  head-mounted  display.  Coincidentally, 
this  also  determines  whether  or  not  subjects  have  a  stationa^  visual  background.  The  rest 
frame  hypothesis  would  predict  that  there  should  be  more  sickness  in  the  blinders-on 
condition.  To  our  dismay,  this  proved  to  be  the  case,  and  we  had  to  change  our  method  of 
blocking  the  subjects’  view  of  the  laboratory. 

5. 2. 2.  Driving  simulator 

We  have  written  a  proposal,  in  partnership  with  a  company  called  Motion  Research,  to 
create  a  driving  simulator  that  does  not  suffer  from  the  problems  of  simulator  induced 
sickness.  The  abstract  is  shown  below: 

The  Auto  racing  simulator  industry  to  date  has  been  primarily  the  preserve  of  the 
entertainment  industry.  While  the  entertainment  industry  continues  to  develop  and  produce 
increasingly  sophisticated  games,  there  is  a  limit  to  the  fidelity  that  they  will  need  or 
accomplish.  Motion  Research  has  identified  a  need  for  a  high-fidelity  auto  racing  simulator 
that  is  more  than  a  game  and  that  can  actually  be  used  to  train  drivers  preparing  for  a  career 
in  the  auto  racing  industry.  World  wide,  auto  racing  is  a  $6  billion  enterprise,  and  it  is 
expected  to  grow  by  more  than  10%  per  year. 
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There  have  been  several  attempts  to  build  high  fidelity  auto  and  auto  racing  simulators, 
most  of  which  have  encountered  the  issue  of  simulator  sickness.  This  is  a  problem  that  we 
intend  to  solve.  MR  has  access  to  the  unit  built  and  owned  by  the  Chrysler  Corporation  to 
test  and  develop  road  noise  and  harshness  damping  solutions.  There  has  been  considerable 
work  done  in  the  field  of  simulator  sickness  with  this  unit,  and  that  information  is 
accessible  to  us.  In  addition  we  intend  to  use  the  skills  and  experience  of  the  UW  HIT  lab 
staff,  and  gain  access  to  their  driving  simulator. 

We  propose  a  cooperative  and  simultaneous  approach  to  building  a  sickness-free  simulator. 
MR  will  begin  the  definition  and  prototype  phase  of  the  actual  simulator,  while  the  HIT  Lab 
conducts  experiments  centered  around  the  “rest  frame  hypothesis”  and  aimed  at  reducing 
the  problem  of  simulator  sickness. 

The  product  we  intend  to  produce  will  meet  several  needs: 

•  Reduce  the  cost  of  testing  for  drivers  moving  into  new  forms  of  racing 

•  Minimize  the  risk  to  man  and  machine  during  these  transitional  periods 

•  Allow  further  development  and  refinement  of  vehicle  dynamic  tests  without  risk 

•  Provide  a  very  realistic  high-end  racing  experience  for  race  sponsor  executives 
and  guests 

•  Develop  a  unit  that  can  be  used  for  initial  driver  training 

Motion  Research  is  uniquely  qualified  to  develop  the  market  and  product.  Its  principal  has 
over  22  years  of  professional  racing  experience  and  cctntact  deep  within  the  auto  racing 
industry.  Other  principals  in  the  company  have  had  experience  in  human  interface  products 
and  technology  and  have  participated  in  many  hours  of  the  type  of  track  testing  that  can  be 
eliminated  by  a  racing  simulator. 

Motion  Research  plans  to  employ  as  many  as  40-50  people  in  the  next  2-3  years  as  a 
Washington  State  company.  These  people  will  be  responsible  for  the  development  and 
operation  of  our  simulators  as  they  are  used  by  racing  drivers  at  various  locations. 

5.3.  Virtual  Motion  Controller  and  spatial  awareness 

5. 3.1.  Spatial  Awareness  Group 

We  have  created  a  working  group  to  conduct  a  literature  review  to  answer  the  question 
“What  is  human  spatial  awareness”.  We  are  planning  to  submit  a  proposal  to  the  ONR  and 
to  NIH  to  fund  some  of  this  research.  ONR  are  interested  in  the  training  aspects  of  spatial 
awareness.  We  are  hoping  to  solicit  support  from  the  NIH  on  the  developmental  aspects  of 
spatial  awareness  -  how  it  is  gained  in  the  young,  and  how  it  is  lost  in  the  old. 

5. 3. 2.  Head  to  Head 

The  VMC  was  licensed  by  a  small  start-up  company  for  commercial  exploitation.  After 
conducting  extensive  market  research  the  company  decided  that  the  best  commercial 
opportunities  were  in  the  simulator  markets,  as  against  the  domestic  computer-game 
markets.  Head  to  Head  is  currently  negotiating  with  a  Government  agency  to  use  the  VMC 
in  a  dismounted  infantry  simulator. 
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